Least Squares Parameter Estimation, Tiknonov Regularization, and Singular Value Decomposition

This handout addresses the errors in parameters estimated from fitting a function to data. Any sample of measured quantities will naturally contain some variability. Normal variations in data propagate through any equation or function applied to the data. In general we may be interested in combining the data in some mathematical way to compute another quantity. For example , we may be interested in computing the gravitational acceleration of the earth by measuring the time it takes for a mass to fall from rest through a measured distance. The equation is d = gt 2 /2 or g = 2dt −2. The ruler we use to measure the distance d will have a finite resolution and may also produce systematic errors if we do not account for issues such as thermal expansion. The clock we use to measure the time t will also have some error. Fortunately , the errors associated with the ruler are in no way related to the errors in the clock; they are statistically independent or uncorrelated. If we repeat the experiment n times, with a very precise clock, we will naturally find that measurements of the time t i are never repeated exactly. The variability in our measurements of d and t will surely lead to variability in the estimation of g. As described in the next section, error propagation formulas help us determine the variability of g based upon the variability of the measurements of d and t. The remainder of this document addresses how errors in parameters , (such as the gravitational constant, g) can be evaluated and reduced by modifying the equations describing the system. We will see that in many parameter estimation problems, the fundamental nature of the problem can lead to parameter errors that are so large that they can make the problem almost impossible to solve. We will see that by modifying the problem slightly the problem will become solveable, but by doing so the resulting parameters may be biased. In this way we will be able to control the trade-off between systematic bias errors and random errors.